Retention
The [Configuration>General Settings >Retention] menu can be used to manage data retention.
The Retention Mechanism allows automated management of data life, considering factors such as performance and data size.
Data streams are stored on partitions of which there are four types in the system:
- Hot - data just written, updated - maximum read and write speeds, high disk utilization.
- Warm - deferred data that is no longer written while they can be read and searched.
- Cold - low read and speed, low disk utilization.
- Delete - data to be deleted.
You can change the storage location of a particular Data Stream in the context of the partition type. The maxTime parameter is used for this purpose and it is the maximum time after which the partition type will be changed to another.
When the maxTime condition is met, the partition type is changed in the following order:
It is impossible to change partitions for a particular data stream from a type with a lower status to a type with a higher status, e.g., Cold -> Warm or Warm -> Hot.
Retention policy
Data retention takes place in the system based on user-configurable Retention policies. It is required to define at least two types of partitions for the policy. Policies are executed periodically in the order specified in the Priority field. The moment of execution is indicated by the Time to next execution field. The value in the Priority field must be within the range of 1-100, where 1 is the highest priority, and 100 is the lowest. When creating policies, care should be taken to ensure that policies do not overlap or duplicate because all defined policies will be executed.
If you operate the maxTime parameter in hourly units, the system since version 2.3 stores data up to 1 hour longer, for example: 1 Hour Hot + 1 Hour Warm means that data is stored minimum 2 hours and maximum 3 hours.
After the installation, the system has four built-in Retention policies. Please check them and adjust them to your needs. They can be used as a template to create new user policies.
If you do not define a Delete partition in the Retention policy the data to which the policy applies will never be erased.
Default Retention policy
1. Built-in policy parameters for the netflow data stream.
| Partition type | maxTime |
|---|---|
| Hot | 1 hour |
| Warm | 1 hour |
| Cold | - |
| Delete | 1 hour |
This means that the data from the netflow stream is stored in the system for 2 hours, within the first hour the data wille be stored with the Hot index, and within the next hour - with the Warm index. In the last hour the data will have the Delete status, which means that only certain metadata will still be stored and it is not possible to restore the original data to the system.
2. Built-in policy parameters for the aggregated data stream.
| Partition type | maxTime |
|---|---|
| Hot | 1 day |
| Warm | - |
| Cold | - |
| Delete | 1 day |
This means that the data from the above-mentioned aggregated streams is stored in the system for 1 day, with the Hot index. On the second day, the data will have the Delete status, which means that only certain metadata will still be stored and it is not possible to restore the original data to the system.
3. Built-in policy parameters for the alerts.
| Partition type | maxTime |
|---|---|
| Hot | 1 week |
| Warm | - |
| Cold | - |
| Delete | 1 day |
This means that the alerts are stored in the system for 1 week, with the Hot index. After one week, the data will have the Delete status for 1 day, which means that only certain metadata will be stored and it is not possible to restore the original data to the system.
4. Built-in policy parameters for audit messages, metrics and notifications.
| Partition type | maxTime |
|---|---|
| Hot | 1 month |
| Warm | 1 month |
| Cold | - |
| Delete | - |
This means that the data is stored in the system for 2 months. First the first one with the Hot index and the second one with the Warm status.
Disk usage
In this section, the table indicates how much space the data takes up in the system.
